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Abstract 


The Russian universities as well as universities from other countries increasingly compete not only 
at the national level, but globally. This trend is reflected in growing interest to universities rankings. 
Despite criticism, rankings outcomes are in demand and influence universities’ positioning in the global 
higher education area. In Russia several rankings have been recently established, striving to satisfy 
needs of various stakeholders. However, all these approaches are single dimensional rankings that use a 
composite indicator and weight coefficients. The article presents a rationale and draft methodology of a 
multidimensional ranking system in Russia. The authors advocate relevancy of the chosen approach as 
it allows them to reflect complexity and diversity of the Russian Higher Education system. Drawing on 
the project outcomes, the authors focus on the national multidimensional ranking methodology concept, 
choice of indicators, the approbation outcomes, dilemmas and decisions. 
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Introduction 


Driven by globalization and growing demands of various stakeholders modern 
universities become more transparent and intensify their engagement with stakeholders. 
Enhancing competitiveness and transparency (van Vught & Westerheijden, 2010; Rauhvargers, 
2011) strengthening image and reputation at national and international levels (Hazelkorn, 2011) 
become key goals for universities’ development. The growing mobility of international students 
and researchers worldwide challenge higher education institutions for better quality provision 
(OECD, 2010). 

Universities are striving for better reputation to attract more talented students and 
researchers and such tools as accreditation, quality assurance and academic rankings play an 
important role in strengthening their reputation and image. 

Despite criticism, rankings are perceived as one of the tools that influence universities’ 
positioning in the global higher education area. However, some experts consider that international 
academic community and governments are obsessed with rankings (Hazelkorn, 2011). 
Nevertheless, rankings serve as a useful tool to define universities’ status and competitiveness. 
A growing number of various rankings and their popularity in Europe and the USA are explained 
by massification, marketisation and globalization of higher education (Shin, & Toutkoushian, 
2011). These trends are also relevant for Russia. 
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To facilitate modernization, enhance higher education quality, strengthen universities 
capacities, the Russian government established federal universities (Ministry of Education 
and Science RF, 2012, retrieved 5 November, 2012, from URL: http://www.mon.gov.ru) and 
national research universities (National Research Universities, 2012, retrieved 5 November, 
2012, from URL: http://eng.mon.gov.ru/pro/ved/niu/). This transformation of the Russian 
Higher Education System led to increased competition and differentiation among universities 
(Drantusova & Knyazev, 2011). 

Simultaneously with the above-mentioned trends the Russian universities are making 
concerted efforts to get top positions in global rankings. The Russian government set a goal for 
the national universities to reach top-positions among global universities by 2020. However, at 
the moment the results are modest: there are only two Russian universities in the Times Higher 
Education World University Rankings (THE) (The Times Higher Education, 2012, retrieved 
5 November from URL: http://www.timeshighereducation.co.uk/world-university-rankings/) 
and 14 universities in Quacquarelli Symonds World University Rankings (QS Ranking) 
(QS Rankings, 2012, retrieved 1 November, 2012, from URL: http://www.topuniversities. 
com/university-rankings/world-university-rankings).Therefore, special systematic efforts are 
required to strengthen the national universities competitiveness in the global higher education 
area. 

Alongside with the increasing popularity of global academic rankings, the number of 
national ranking systems, monitorings and quality assessments are growing. National policy in 
higher education is significantly affected by outcomes of the monitorings implemented at the 
initiative of the Ministry of Education and Science RF (National Training Foundation, 2012, 
retrieved 5 November, 2012, from URL: http://ranking.ntf-ru/). 


Problem of Research 


Rankings can be developed by media, professional associations, and universities. The 
quality of a ranking system is responsibility of its developer. There is no single organization that 
assesses rankings quality. The International Observatory on Academic Ranking and Excellence 
(IREG) is an independent expert organization that conducts an independent rankings’ audit 
and provides recommendations (IREG, 2012, retrieved 5 November, 2012, from URL: http:// 
www.ireg-observatory.org/). In Russia there have been developed more than 30 tools of higher 
education institutions’ external evaluation. They draw close attention of prospective students, 
universities and academic community. They are also criticized by various stakeholders 
(Rauhvargers, 2011). None of these ranking systems undertook the IREG audit procedure. This 
could be explained by the fact that these ranking systems have recently become the focus of 
attention. 

A comparative analysis of 19 international and 30 Russian ranking systems was 
carried out within the framework of the project “Developing and Approbation of a Template 
Methodology for National Ranking of Higher Education Institutions” (2011-2013) implemented 
by National Training Foundation. The ranking systems were assessed against IREG audit criteria 
(Zavarykina, Lopatina, & Perfilieva, 2012). The analysis revealed the number of limitations of 
the national ranking systems including: 

Inconsistency: some new rankings appear to meet immediate demands and are not repeated 
by developers due to lack of institutional, managerial, financial and human resources. 

Diversity of target audiences and ranking developers: some rankings are developed to 
meet demands of the national education system and are initiated and implemented by the Ministry 
of Education and Science RF; other rankings are initiated by media, business community and 
are aimed at meeting demands of general public; 


ISSN 1822-7864 


PROBLEMS 
OF EDUCATION 


IN THE 21* CENTURY 
Volume 48, 2012 





PROBLEMS 
OF EDUCATION 


IN THE 21* CENTURY 
Volume 48, 2012 





Institutional focus: existing ranking systems usually assess higher education institutions 
(HEI) but not education programmes; 

Limited assessment: only separate functions of higher education institutions are 
assessed (usually teaching and research). 

Acomparative analysis demonstrated that the Russian ranking tools do not fully comply 
with the audit criteria of the International Observatory on Academic Ranking and Excellence 
(IREG). Crucial limitations of the Russian ranking systems are: 

e = Lack of transparency; 
e Lack of feedback on rankings outcomes from main stakeholders; 
e Quality of the received data. 

The limitations drawn from the analysis determine the need to elaborate a new approach 
to the academic rankings in Russia. The new approach should provide the transparent and 
reliable information about the national higher education system functioning. The methodology 
of the new academic ranking must consider the diversity and development trends in national 
system of higher education as well as to comply with international quality criteria and standards 
for ranking development. The multidimensional approach is suggested as the most relevant to 
address all the requirements. 

Consider limitations of the existing ranking tools in Russia the correct assessment and 
comparison of the national higher education institutions are currently challenged, though there 
is a great demand for the objective and integrated assessments for the evidence-based policy 
making in Russia. Thus, the research project was designed to identify the key elements of the 
methodology for the multidimensional academic ranking and to approbate it for the national 
higher education system. 


Research Focus 


The Russian higher education system is highly diversified and presented by a number of 
following groups and types of HEIs, including: 

e leading universities (Moscow State University, Saint Petersburg University, federal 
universities, national research universities); 

e other universities (e.g., Tver State University; Ryazan State University); 

e engineering and technical higher education institutions (e.g. Irkutsk Technical 

University); 

e humanitarian and pedagogical higher education institutions (e.g. Russian Humanitarian 

University); 

e higher education institutions with economics and law programmes (e.g. Rostov State 

Economics University); 

agricultural higher education institutions (e.g. Orlov State Agricultural University); 

medical higher education institutions (e.g. St. Petersburg Medical Academy). 

The existing Russian ranking practices solve narrow objectives and do not take into 
account differentiation and complexity of the Russian Higher Education System as well as do 
not facilitate integration of the Russian higher education institutions into the global education 
area. The Russian rankings also do not fully comply with the international criteria of rankings’ 
quality. 

An integrated assessment of higher education institutions is required to meet new 
challenges of international and national higher education systems development. The presented 
article describes a new approach to assessment of national higher education institutions on 
the basis of multidimensional ranking that enables to meet objectives of the national higher 
education system development. 
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Methodology of Research 
General Background of Research 


The aforementioned project goal is to develop and approbate a template ranking 
methodology of the Russian higher education institutions on the basis of: 


e the methodology of comparative analysis of global, national and specialized rankings, 

national approaches to evaluation of higher education institutions performance; 

e public and expert discussions; 

approbation and evaluation of the outcomes; 
consultations with experts from the International Observatory on Academic Ranking 
and Excellence. 

The methodology is based on a multidimensional ranking. 

The selection of a multidimensional approach to ranking higher education institutions 
responds to the diversity of the national higher education system and necessity to take into 
account various dimensions of higher education institutions activities. The developed 
methodology also takes into account the best international practice that was identified through 
a comparative analysis of foreign and international ranking methodologies and their assessment 
against IREG audit criteria. (Table. 1) 

15 parameters (type of ranking, focus of a ranking, target groups, subject areas, status, 
frequency, geographical scope, requirements to the participants, education levels, methodology, 
outcomes, criticism, reputation, position of the Russian universities) have been used for 
comparative analysis of rankings systems. These parameters enable a common framework for 
the comparison (Zavarykina, Lopatina, & Perfilieva, 2012). 


Table 1. Characteristics of compared foreign and international ranking method- 
ologies. 





Type 


of methodology Characteristics Methodologies 





Academic Rankings of World Universities. (2012). 
Retrieved 1 November, 2012, from URL: http://www. 
shanghairanking.com/ 

The Times Higher Education World Universities Rank- 
ings. 2012. Retrieved 5 November from URL: http://www. 
timeshighereducation.co.uk/world-university-rankings/ 


Displays vertical diversity in terms 
of performance by using indicators. 
Most existing higher education 
rankings take the form of a league 


Single-dimen- fable QS World University Rankings. 2012. Retrieved 1 
sional ranking November, 2012, from URL: http://www.topuniversities. 
(rankings, com/university-rankings/world-university-rankings 


League table — a single-dimensional 
list going from “best” to “worst’, 
assigning ordinal numbers to the 
entities which relate only to rank 
and scales of difference. 


US News Worlds Best Universities Ranking. (2012). 
Retrieved 1 November, 2012, from URL: 
http://(www.usnews.com/education/worlds-best-universi- 
ties-rankings 

Guardian Ranking. (2012). Retrieved 1 November, 2012, 
from URL: 
http:/www.guardian.co.uk/education/universityguide 


league tables) 
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D ttry t bine educat : ; 
oes not iy to comoine education | U-multirank. van Vught, F.A.& Ziegele, F. (2011) 
and research rankings, for example, 


inte-sindle: composite measure Centre for Higher Education (CHE) University Rankings. 
sie Aabncsniaaes (2012). Retrieved 1 November, 2012, from URL: http:// 

and is often user-driven because : ee ui 

: , ioe www.che-ranking.de/cms/?getObject=613&getLang=en 

it enables an interactive display of 


data. 


Multidimension- 
al ranking 





Asystem that allocates objects 

to groups on the basis of their 
characteristics. It shows horizontal 
diversity, where differences do not 
imply ordinary scales of “more”, 
“bigger” or “better”. It is aimed at 
showing diversity of higher educa- 
tion institutions. 


European Classification of Higher Education. U-map. 
(2012). Retrieved 1 November, 2012, from URL: http:// 
Www.u-map.eu/ 

Classification 
The Carnegie Classification of Institutions of Higher Edu- 
cation. (2012). Retrieved 1 November, 2012, from URL: 
http://classifications.carnegiefoundation.org/ 

















The comparative analysis includes 3 levels: 

I level: Comparative analysis of methodologies on key selected parameters 

2 level: Assessing ranking methodologies against the IREG audit criteria 

3 level: Identifying key quantitative indicators and assessing these indicators against 
criteria of relevance to the Russian education system development objectives, validity and 
feasibility of data collection. 

The assessment of the analysed methodologies against the IREG audit criteria 
demonstrated that multidimensional rankings such as Multirank, CHE University Ranking have 
greater compliance with the criteria. However these approaches have their limitations: 

e Resource intensity; 

e Lack of comparable data on HEIs performance; 

e Challenges of ensuring data validity received from surveys; 

e Challenges of ensuring data collection procedures validity and quality in case of 

large volume of data collecting; 

e Difficulties associated with processing of large volume of data; 

e Challenge of covering all subject areas. 

Despite these limitations the multidimensional ranking enables to design a new approach 
to assessment and ranking of higher education institutions. It considers various functions and 
types of national higher education institutions and enables to conduct accurate assessment 
and comparison of higher education institutions. Following the outcomes of the conducted 
analytical research a multidimensional approach became a basis for assessment and ranking of 
the national higher education institutions. 

Key principles of the template methodology for ranking Russian higher education 
institutions build on the features which should a new approach to HEIs assessment should 
conform to: 

1. The methodology should provide reliable data on performance of higher education 
institutions and their position in system of higher education; 

2. The methodology should take into account diversity of the Russian higher education 
institutions, their missions and functions; 

3. The methodology should be a useful tool for users of educational services providing 
friendly and easy-to-use-and-interpret information on diversity of higher education 
institutions and education programmes; 

4. The methodology should take into account diversity of education services users; 

5. The methodology should facilitate quality improvement and competitiveness of higher 
education institutions; 
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6. The methodology should facilitate integration of the Russian higher education 
institutions into global education and research environment as their position in rankings 
is perceived as a “signal” of universities competitiveness; 

7. The methodology should be a source of valid data for global and regional rankings. 

Five functions of higher education institutions have been identified and included in the 
ranking methodology: research, teaching and learning, internationalization, knowledge transfer, 
engagement with regional stakeholders. The methodology for ranking Russian higher education 
institutions takes into account information needs of the following several target users groups: 

1. Prospective students and their parents; 

2. Government (central and local); 

3. Employers and labour market; 

4. Academic community (researchers, lecturers); 
5. Business organisations. 

A multidimensional approach, used in the methodology, is aimed at achieving maximum 
relevance in HEIs assessment and correctness in their comparison. Under a multidimensional 
approach an aggregated score is not used to assess and/or compare HEIs. 

Multidimensionality of the methodology is based on confluencing several evaluation 
areas (figure 1): 

e Five identified HEIs functions: teaching, research, knowledge transfer, engagement 
with regional stakeholders, internationalization. 

e HEIs’ groups identified according to their missions: leading universities (Moscow State 
University, Saint Petersburg University, federal and national research universities), 
other classical universities, engineering and technical HEIs, humanitarian and 
pedagogical HEIs, HEIs with economics and law programmes, agricultural and medical 
universities. 

e Groups of users identified according to their information needs: prospective students, 
academic community, government, business organisations. 
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Figure 1: Multidimensionality of a new academic ranking methodology. 


As outlined in the key methodology principles the mission of the template methodology 
is to provide reliable and objective information on HEIs performance to satisfy information 
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68| needs of various groups of education services users with account of the Russian higher education 
system diversity. 

The main goals of the template methodology are, as follows: 

Assessment of higher education institutions against 5 identified functions. 
Developing a database of the Russian higher education system (current state and 
development trends) taking into account its diversity, with possibility of creating HEIs 
rankings and ratings on specifies indicators. 

e Developing a transparent tool for external assessment of the Russian higher education 
institutions. 

e Contributing to the Russian higher education system development through creating an 
information and analytical basis for benchmarking (best practices identification) and 
facilitating demand for higher education services in the country. 

The multidimensional approach complies with the methodological standards for rankings 
development: 

e standards for empirical research; 

e rankings quality criteria developed by the international expert group on Academic 
Ranking and Excellence (IREG) and Berlin principles on ranking higher education 
institutions; 

e practices on developing global, national and specialist rankings. 

The developing approach does not use qualitative data received from surveys of students, 
academic staff and employers. This is explained by the lack of specialized national sociological 
research. However, inclusion of qualitative data into the multidimensional ranking methodology 
is perceived as one of the directions of the Russian higher education system development. 

All elements of the multidimensional ranking methodology (in the first instance, 
quantitative indicators) have been discussed and evaluated by experts. 

Experts’ inputs have been used for: 

e indicators identification for each of 5 functional areas of assessment with account of 
international and Russian practice; 

e assessing indicators against the criteria of relevance to the Russian higher education 
system development objectives; their validity; availability; relevance to the 
methodology; 
testing the template ranking methodology; 
weighting indicators within each functional area. 

On the basis of the experts’ assessment 65 indicators on 5 HEIs’ functions (research, 
teaching and learning, internationalization, knowledge transfer, engagement with regional 
stakeholders) have been identified. 





Sample of Research 


There is more than a thousand of higher education institutions in the Russian Federation 
(Statistics in Education, 2012, retrieved 1 November, 2012, from URL: http://stat.edu.ru/scr/ 
db.cgi?act=listDB&t=2_6 la&ttype=2&Field=All). To construct the sample structure for the 
approbation of the developed multidimensional approach to academic ranking 10 per cent of 
higher education institutions were selected out of the general totality of HEIs in national system 
of higher education. The sample structure takes into account the existing typology of national 
HEIs and reflects the geographical distribution of HEIs across the country (Yadov 2007, 95 
- 105). 

Thus, 148 higher education institutions have been invited to participate in approbation 
of the methodology. 

The sample structure for the approbation includes: 
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e Leading Russian universities (Moscow State University, Saint Petersburg State 
University); 

e Federal universities (9 universities); 

National research universities (29 universities); 
Higher education institutions which received government support of their strategic 
development programmes (54 HEIs); 

e Higher education institutions which education programmes have been listed as the 
best educational programmes (catalogue 2011 — 2012 “Best education programmes: 
innovation Russia) (42 HEIs); 

Higher education institutions recommended by experts (10 private HEIs); 
Higher education institutions which expresses interest in taking part in approbation 
(3 HEIs). 

Selected HEIs demonstrate different missions. Leading national universities such as 
Moscow State University and Saint Petersburg State University are the acknowledged world 
leaders of higher education preserve traditions of academic universities with balanced functions 
of teaching and research. 

As well as the Federal universities (FU) which are also aimed to provide high quality 
education and research but with a special focus to the needs of social and economic development 
of their regions. Each of federal universities functions in one of nine federal districts. 

National Research Universities (NRU) are not affiliated to a special region. There are 29 
research universities each of which is aimed at developing applied sciences to meet the needs 
of a particular industrial sector (aviation, space navigation, chemicals etc.). 

Among other higher education institutions chosen for the approbation different types of 
higher education institutions presented. They also have a special vision. 

HEIs which received government support of their strategic development programmes 
(ME) are committed to a holistic approach to university development and focused on education 
quality, international cooperation and networking with regional stakeholders. Most of these 
HEIs are regional universities, specialized in engineering, technical, pedagogic or social 
sciences programmes. The Peoples’ Friendship University of Russia could be named here as 
an example. 

Another type of higher education institutions presented in the sample are HEIs with the 
best innovative educational programmes recognized by the National Centre for Professional 
Accreditation (The Best Educational Programs of Innovate Russia, 2011). 

10 privately funded institutions were recommended by experts to be included into the 
sample structure. These universities are also members of the National Association of private 
HEIs. 

The main feature of those HEIs which expressed interest in taking part in approbation 
is that they offer special educational programmes in Technology and Engineering and usually 
are not covered by national academic rankings. These institutions, however, are interested 
in comparing themselves with other universities. The approbation is, therefore provide these 
universities an opportunity to attract additional attention to their special programmes. 

Thus, this approach enables to include various types of education institutions into the 
approbation of the methodology. 

The data received within the approbation is used as the base for a pilot ranking of Russian 
higher education institutions on 5 functions: research, teaching/learning, internationalization, 
knowledge transfer, engagement with regional stakeholders. 
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graphical distribution. 


70| Table 2. Sampling structure of the approbation by types of HEIs and their geo- 
































HEI with 
Federal Classical Techni- Pedag. Economics/ Medi- Ag- TO- 
District universi- cal HEI Law cal ricul. TAL 
aut ties HEI pro- HEI HEI 
grammes 
1 North-Western 7 8 3 3 1 22 
2 Central 9 17 8 5 2 1 42 
3 Volga 9 12 3 2 1 2 29 
4 Southern 3 4 2 1 10 
5 North-Cauca- 5 ' 1 7 
sian 
6 Ural 3 3 1 7 
7 Siberian 5 7 4 3 2 1 22 
8 Far-Eastern 6 2 1 9 
TOTAL 47 54 19 17 7 4 148 
































All 148 universities presented in the sample were invited to the approbation of the 
methodology. 103 institutions (A leading university; 8 federal universities; 28 national 
research universities; 28 higher education institutions which received government support of 
their strategic development programmes and 38 other universities) agreed to take part in the 
approbation of the methodology. The sampling structure was saved, but agricultural universities 
refused to participate in the project. As the participation in approbation was voluntary the 
agricultural universities claimed that they do not have capacities to fill in the questionnaire and 
provide the data. 

The approbation took place from April to July 2012. 


Instrument and Procedures 


A special toolkit was developed to carry out the approbation. The toolkit included the set 
of indicators and the questionnaire as mentioned below. 

The set of indicators consists of 65 indicators divided into six groups. These groups 
include: 

Group A «Higher Education Institution Profile» 

Group B «Research» 

Group C «Teaching and Learning» 

Group D «Internationalization» 

Group E «Knowledge Transfer» 

Group F «Engagement with Regional Stakeholders» 

Each group has its internal structure and content (National Training Foundation, 2012, 
retrieved 5 November, 2012, from URL: http://ranking.ntf.ru/) 


Group A — Higher Education Institution Profile (Al — A19) 


This group consists of indicators providing general information on HEI. This information 
includes indicators such as HEI type (A2), category, group (A3, A4), affiliation (A5), 
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organisational and legal form (A6), HEI geographical location (A7) and territorial belonging 
(A8). Number of students (A9), number of academic staff (A10), number of researchers 
(A13), number of non-academic staff (A11) are also included in HEI profile. This group of 
indicators enables to define HEI’s capacities for education services provision. Information on 
undergraduate education programmes (A12), and on opportunities for postgraduate education 
(PhD and doctorate programmes) (A14, A15) is added to provide more detailed information 
on HEI profile. Other information useful for users of education services can be added to HEI 
profile and the indicators’ list could be expanded. 

Specific characteristic of group A indicators is their functionality. On the one hand each 
indicator has its own meaning and can be used individually and not for assessment purposes, 
on the other hand some of these indicators are included in the structure of indicators from the 
groups (B, C, D, E, F) and can be used for calculations of indicators from these groups. 

Data for group A indicators can be received from documents available in open access. 
The Federal law on “Education” specifies types and categories of higher education institutions. 
The RF Constitution defines the structure of the federal executive structures. A HEI’s affiliation 
is specified by HEI’s foundation documents. Organisational and legal forms of higher education 
institutions are described in the classification system developed by the Ministry of Education 
and Science of the RF. The territory belonging of HEIs can be identified by the Presidential 
Decree N 849 from 13 May 2000 “President’s Representative in the Federal District” and the 
RF Constitution. 

More detailed information on HEI profile can be obtained from the system that 
collects statistical information on Higher education institutions (HPE-1), and data provided 
by universities for accreditation purposes. Information on Federal universities and National 
Research universities can be received from the universities’ development programmes. However, 
existing databases have limitations. Indicators definitions differ as various information and 
monitoring systems have different goals. Therefore, it is not possible to substitute indicators used 
for ranking purposes by indicators from these external systems. Though, the data from external 
databases are used to supplement and enhance reliability of the data used in the rankings. 

Data sources for group A indicators are, as follows: 

e Data provided by HEIs for accreditation purposes; 
e HEIs’ internal data. 


Group B — Research (B1 — B15) 


Indicators from the group B describe HEIs research performance. Both input and output 
indicators are included in the group. 

The indicators of academic staff quality (B2), publication activity (B 12), HEI’s 
participation in grant programmes/projects (B13 — B14) are traditionally used for assessment 
of HEI’s research activity. International practice demonstrates that one of reliable methods to 
assess HEI’s research potential is assessment of HEI’s publication activity, its expenditure on 
research and research income. 

Data sources for the group B indicators are: 

e Data provided by HEIs for accreditation purposes; 
e HEIs’ internal data; 
e Bibliometric databases. 


Group C — Teaching and Learning (CI — C18) 


Group C indicators assess HEI’s performance in learning and teaching. 
This dimension can be characterized by quality of applicants / entrants (C3, C14), 
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quality of students (C4), quality of academic staff (C9, C10, C12, C16), quality of education 
programmes (C1, C2), graduates employability (C5), HEI expenditure on education services 
(C11, C18), opportunities for postgraduate education (C7, C13), and students mobility (C8). 
Russian and international practice demonstrate that assessment of applicants quality, quality of 
academic staff, expenditure on education services and graduates employability are one of the 
most frequently used indicators to assess HEI’s teaching and learning. 
Data sources for the group C indicators are: 
e Data provided by HEIs for accreditation purposes; 
e HEIs’ internal data. 


Group D — Internationalization (DI — D12) 


Group D indicators characterize HEI’s internationalization. The quality of 
internationalization is assessed by indicators describing HEI’s activity in attracting international 
students and lecturers (D1, D5, D12), students’ and lecturers’ mobility (D6, D7, D8), income 
from international sources (D2, D3, D4), and implementation of education programmes in 
collaboration with international partners (D10, D11). Both input and output indicators are 
included in this group. Thus, they enable to assess HEI’s internationalization from different 
points of view. 

Data sources for the group D indicators are: 
e Data provided by HEIs for accreditation purposes; 
e HEIs’ internal data. 

Data for federal and national research universities can be collected from universities’ 

development programmes. 


Group E — Knowledge Transfer (El — E5) 


Group E indicators characterize HEIs activity in transfer of their knowledge to main 
stakeholders. HEI’s performance in this area can be assessed by its economic activity (E1 — E3) 
and HEI’s collaboration with external partners (E4, E5). It should be noted that the indicators 
used to assess this dimension of HEI’s activity are insufficiently developed both in Russian 
and international practice. Data collecting on these indicators can be problematic. However, 
such indicators as income from non budgetary sources and income from intellectual property 
products are widely used both in Russian and international practice. 

Data for the group E indicators can be obtained from: 
e Data provided by HEIs for accreditation purposes; 
e HEIs’ internal data. 


Group F — Engagement with Regional Stakeholders (F 1 — F4) 


The indicators from group F describe HEI’s engagement with regional stakeholders. 
The indicators on HEI’s economic activity in the region (Fl, F2) and the indicators on training 
specialists for the region (F3, F4) enable to assess HEI’s performance in this dimension. 
Data for group F indicators can be collected from: 
e Data provided by HEIs for accreditation purposes; 
e HEIs’ internal own data. 
The questionnaire for approbation includes 6 semantic blocks representing various 
aspects of HEIs’ activities, including: 
e Higher education institution profile (general information); 
e Undergraduate and postgraduate students; 
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Academic, non-academic staff and researchers; 

Education programmes; 

Bibliometry; 

Budget. 
6 blocks of indicators have been developed as the result of decomposition of 65 initial 
indicators identified by experts’ assessment (expert panel consists of 17 experts represented 
HEIs, employers and representatives from the Ministry of Education and Science, RF). Most 
of these primary indicators present relative values (ratio, %) assessing respective functions of 
higher education institutions. 

The questionnaire was constructed to collect separate absolute subindicators. Absolute 

sub-indicators data were used to calculate the 65 indicators. 


Data Analysis 


Standard mathematical and statistical procedures (normalization, aggregation) were 
employed for rankings construction (Antonov, 2004, OECD, 2008). Also, the received raw data 
from the universities were verified, processed and analysed. 

The raw data verification revealed that some institutions were unable to 
provide data on some sub-indicators (primary indicators). Some universities do not 
collect data on specific indicators (nominal data absence) but some universities do 
not have outcomes on a certain activity which is reflected by a sub-indicator (actual 
data absense). 

The absense data were restored by the following steps: 

I step: the missing data were restored from open sources (namely bibliometric 
databases). 

2 step: some data are not available in open sources and can be received only from HEIs. 
Participated universities received a special form with missing data on some indicators and they 
have an opportunity to double check the data and complete the missing data. As mentioned 
above some universities were able to present additional information on missing data, but some 
universities were unable to do so. 

To receive groups of higher education institutions that demonstrated high, middle and 
low performance the statistical method of data grouping was used. The next formula was applied 
to calculate intervals: 
= max —Xmin 

n 
i— interval length, x and x, — maximum and minimum of a grouping characteristic, n — 
number of groups. For research purposes the sample was divided into 3 groups (universities 
- leaders, universities — competitors, catching up universities). 


Results of Research 
The ranking of 103 HEIs enables to identify the following trends. 


e =©Overall ranking of 103 HEIs indicates that 7 National Research Universities (NRU) are 
leaders among all universities (see table 3); 
The group of leaders is homogeneous and consists of National Research Universities; 
The group of “competitors” is heterogeneous and consists of HEIs from various 
categories (National Research Universities (NRU); Federal universities (FE); HEIs, 
which received government support of their strategic development programmes (ME); 
and other universities (other); 
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e =The “catching up” group mostly includes HEIs, which received government support of 
their strategic development programmes, and HEIs from the category “other”. However, 
8 National Research Universities and 8 Federal universities are also presented in this 














group. 
Table 3. Groups of higher education institutions according to the rankings out- 
puts. 
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Competition for leading positions among National Research Universities, Federal 
universities and universities, which received government support of their strategic development 
programmes, is strong. 

Ranking of 103 HEIs on 5 functions of universities enables to conduct a thorough analysis 
of universities activities and identify their weak and strong areas. The obtained results allow 
researchers to compare groups of HEIs against maximum normalized score (see Table 4). 


Table 4. Distribution of HEIs from 4 categories in 3 groups according to the 
ranking outcomes. 






































Groups/ Knowledge Engagement with 
function Research Teaching Internationalization Transtar regional stakehold- 
areas ers 
3 NRU 5 NRU 2 NRU 3 NRU 4NRU 
(ade; 0 FU 1FU 1 FU 1FU 1FU 
0 ME 0 ME 0 ME 1 ME 0 ME 
0 Other 0 Other 0 Other 0 Other 1 Other 
Total 3 6 3 5 6 
8 NRU 14 NRU 4 NRU 14 NRU 13 NRU 
Competitors 1 FU 4FU 0 FU 5 FU 5 FU 
OME 8 ME OME 10 ME 22 ME 
1 Other 6 Other 3 Other 16 Other 23 Other 
Total 10 32 1 45 63 
18 NRU 10 NRU 23 NRU 12 NRU 12 NRU 
Catching up 7 FU 3 FU 7 FU 2FU 2FU 
28 ME 20 ME 28 ME 17 ME 6 ME 
37 Other 32 Other 35 Other 22 Other 14 Other 
Total 90 65 93 53 34 
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Leaders 


The first group of leaders is characterized by high performance in such function areas as 
“Research” and “Teaching and learning”. 

7 leaders also demonstrate high performance in “Internationalisation”. However their 
results in “Knowledge Transfer” are modest and do not achieve maximum. 

The results of the leaders in “Engagement with regional stakeholders” are also not so 
high. Universities from other categories demonstrate high results in this dimension area. 

The universities demonstrated leading positions in rankings have a long tradition in 
teaching excellence. They also established strong links with research centres and industry 
that allows them to conduct cutting edge research. Thus, their high results in “Research” and 
“Teaching and Learning” are not surprising. However, the ranking revealed that such functional 
areas as “Knowledge Transfer” and “Engagement with Regional Stakeholders” are neglected 
in these universities. 


Competitors 


The second group is represented by 14 National Research Universities, 4 Federal, 4 
universities, which received government support of their strategic development programmes 
and | university from the category “other”. 

This group demonstrates high results in “Teaching and learning” and modest results in 
“Research”. Modest results in “Research” could be explained by the fact that most Russian 
universities were mostly focused on teaching in previous years, thus they do not have enough 
capacities to conduct research to meet new requirements of the modernized higher education 
system. It is expected that they will enhance this capacity in this area in the nearest future. 

The results of these institutions in “Internationalisation”, “Knowledge Transfer” and 
“Engagement with regional stakeholders” are not so high. This is also could be explained by 
current transformations within the system of higher education and its ongoing modernization. 
Thus the progress is expected 

Traditionally, Russian universities have been excluded from the international education 
area. They do not have enough practice in engagement with international partners. However, 
there are a lot of positive trends in this area at the moment. 


Catching up universities 


The third group of universities consists of 73 universities. The most universities in this 
group are presented by “other” universities (34 universities). However, there are 8 National 
Research Universities and 4 Federal universities in this group. 

This group of universities is characterized by high performance in “Knowledge Transfer”. 
The weakest area of these universities is “Internationalisation”. 

It is seen that targeted efforts on internationalisation of education and research in 
these universities will enable to enhance teaching and research and competitiveness of these 
universities. 
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Conclusions 


The developed methodology enables accurate assessment of universities with accounts 
of various HEIs functions. 

The indicators used in the methodology meet the requirements of the Russian Higher 
Education system development objectives and facilitate promotion of the national universities 
into the global higher education area. 

National research universities are leaders among 103 HEIs participated in the 
approbation. 

There is strong competition among national research universities, federal universities and 
higher education institutions which received government support of their strategic development 
programmes for leading positions. 

Most universities should enhance their research capacities (especially publication 
activity) and strengthen their efforts on internationalisation. 

The template methodology developed within the project “Developing and Approbating 
a Template Methodology for National Ranking of Higher Education Institutions” lays the 
groundwork for a national approach to performance evaluation of national higher education 
institutions with account of the Russian Higher Education system diversity. 

The approbation demonstrates that the developed multidimensional ranking methodology 
enables to reveal qualitative characteristics of the Russian higher education institutions; 
identify their strengths and weaknesses on 5 function areas, compare HEIs performance within 
categories. Thus, multidimensional assessment allows researchers to conduct a qualitative 
analysis of a university development and to draw up sound recommendations based on the 
outcomes of this analysis. This enhances capacities of the Russian higher education institutions 
both at national and global higher education area. 


Implications for Practice 


On the basis of the current study, the following practical steps were suggested: 

e Amendment of the methodology by reducing the number of indicators (namely input 
indicators), clarification of definitions for some indicators, developing and clarification 
the system of weights for indicators and areas of assessment. 

e Use the ranking outputs for universities strategic development by creating special 
profiles for every university participated in the approbation. The profile will include 
analysis of HEI strengths and weaknesses and HEIs position on key indicators and 
function areas. 

e The amended methodology should undertake the IREG audit procedure. 

The multidimensional ranking should be done every year with the support of the 
Ministry of Education and Science. 

e It is suggested to add qualitative assessment of universities on the basis of academic 
staff, students and employers surveys. 

e Discuss an opportunity to develop a regional system of HEIs assessment with the 
inclusion of universities from the Eurasian Economic Community and Moldova, 
Ukraine, Armenia. 

e In partnership with the IREG experts identify “core indicators” that are relevant both 
for foreign and Russian universities and various stakeholders. 
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Dilemmas of a Template Methodology 


While designing a multidimensional ranking methodology the researchers face the 
following dilemmas: 

1. National higher education system is highly differentiated which complicates comparison 
of different HEIs. 

2. Identifying relevant indicators for various types of national HEIs is a challenging task. 
Not all indicators are completely relevant for various types and groups of Russian 
higher education institutions. 

3. Various methods are applicable for procedures of aggregating and weighting of 
indicators. The large number of basic indicators, HEIs differentiation lead to dispersion 
of assessment scores and complicate the procedure of indicators weighting. 

Possible approaches to address these dilemmas were suggested. 


Dilemma 1. National higher education system differentiation 


Choices: 
e to include all groups and types of higher education institutions into the methodology; 
e to limit coverage of higher education institutions: do not include specialist institutions 
(with art, physical education and sport programmes), private and municipal HEIs into 
the methodology. 
Further amendment of the methodology will enable: 
to compare HEIs within separate groups 
by separate functions 
by tailored indicators 
by aggregate indicators; 


Dilemma 2. Indicators relevance 


Choices: 

e to use the full set of indicators for all types and groups of higher education 
institutions; 

e to identify “core” indicators that are relevant for all types and groups of HEIs and 
specific indicators that are relevant for separate HEIs groups; 

Further amendment of the methodology will enable: 

e to compare all HEIs by the full set of indicators; 

e to compare all HEIs by a separate set of indicators (core and specific indicators); 

e to compare separate HEIs groups by the full set of indicators; 

e to compare separate HEIs groups by tailored set of indicators. 


Dilemma 3. Aggregation and indicators weighting 


Choices: 

e to increase the number of experts involved in indicators weighting before obtaining the 
statistically significant results of experts assessments; 

e tomake the weighting procedure more complex to develop a separate system of weights 
for separate HEIs groups and separate HEIs functions; 

e to aggregate only homogeneous or close indicators within single area of evaluation. 

Further amendment of the methodology will enable: 
e to compare single area of evaluation by aggregated indicators; 
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e to compare separate HEIs groups by aggregated indicators. 
The next step of the methodology development will be its ammendment with account of 
the approbation outputs and experts consultations. 
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